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METHOD AND APPARATUS FOR 
MANAGING STREAMING DATA 

CROSS REFERENCE TO RELATED APPLICATIONS 

This application claims priority from provisional patent application 60/157,998 
entitled STREAMING MACHINE-AWARE BINARY AND MULTIMEDIA CONTENT 
FOR MULTIMEDIA SYNCHRONIZATION filed 1 0/06/99. 

The above referenced application(s) are incorporated herein by reference for all 
purposes. These prior applications, in some parts, may indicate earlier efforts at 
describing the invention or describing specific embodiments and examples. The present 
invention is therefore best understood as described herein. 

FIELD OF THE INVENTION 

The invention generally relates to methods and/or devices related to data 
transmission and presentation using a computing device or information appliance. More 
particularly, the invention in various aspects relates to multimedia data methods and 
applications and to a versatile file structure suitable for providing multimedia data and 
streaming data over a communication data network. 

BACKGROUND OF THE INVENTION 

Previous works related to aspects of the present discussion include below listed 
reference by Blakowski et al. and by the WWW Consortium. The Blakowski reference 
provides a survey of research regarding media synchronization. The reference discusses a 
four-layer synchronization reference model. From the lower layer to the upper layer are 
Media Layer, Stream Layer, Object Layer, and Specification Layer. A variety of 
techniques have been published and implemented in the specification area. SMIL is one 
of the latest efforts in the industry to provide a presentation specification using 
descriptive tagged language, which follows the syntax of XML. 

Existing and popular media container formats for computers include the 
QuickTime from Apple Computer (http://www.apple.corn), ASF/AVI from Microsoft 
(http://www.microsoft.com), RealMedia from RealNetworks (http://www.real.corn), and 
Video Capsule from Digital Lava (http://www.digitalIavacorn). Some of these such as 
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RealMedia and Video Capsule do not have published technical specifications. Some 
existing formats, such as .exe video files, provide a means for distributing media clips that 
are executable and viewable in a standard operating system without the need for installing 
additional software. In such a file, necessary software components are included in a 
5 single file with a name like movieclip.exe. Generally, such a file is not capable of playing 
streaming data or of varying the components downloaded. 

Downloadable machine code technology (e.g. applets) is also known. Sun 
pioneered aspects of downloadable executable components by creating Java, which has 
become used to make software components downloadable through web browsers to run 
10 on a local virtual machine. Microsoft's COM technology is a later developed technology 
ready to be downloaded from Internet and run in a document container, such as a web 
browser. 

The Internet is a rapidly growing communication network of interconnected 
computers around the world. Together, these millions of connected computers form a vast 

15 repository of hypermedia information that is readily accessible by users through any of 
the connected computers from anywhere and anytime. As there are an increasing number 
of users who are connected to the Internet and surf for various information, there is 
tremendous demand for more content to be available and methods to deliver content on 
the Internet. Currently, the most commonly available media information that is 

20 deliverable on the Internet may include text information, images and graphics, videos and 
audio clips. 

One known format for this data is streaming data, often used for video and/or 
audio data. For example, streaming video is a sequence of encoded images sent over a 
network or other communications media and presented at a user device, with presentation 

25 of part of the data overlapping some of the receiving of the data. With streaming data, a 
user does not have to wait to completely download a large file before accessing that file 
or parts thereof (e.g. seeing video or hearing sound). As will be understood in the art, 
streaming data, such as video data, can be "live" data, such as streaming encoding of a 
radio or television broadcast, or can be previously stored data, such as pre-recorded audio 

30 or video data. The designation "real-time" is sometimes used in the art to indicate 
streaming data generally, included pre-stored media components. This term is used to 
indicate that delivery of the data is time-critical in that once playback begins, the 
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streaming data must be delivered quickly enough to keep up with "real-time" presentation 
of the data to the user. Such data is sometimes also referred to in the art as "clocked 
data" to indicate that the data generally must be delivered at a nearly constant rate. Too- 
fast delivery can swamp the receiving end's buffer capacity. Too slow delivery will 
cause unacceptable halts or pauses in playback. 

To access streaming data, a user generally uses an access module. This module 
can be either software or hardware, with logic functions able to present the data in a form 
desired by a user. As examples, an access module could be a software player (such as 
RealMedia™) running on a compatible information platform. An access device may also 
be more dedicated hardware, such as a cell-phone capable of accessing certain types of 
streaming data, a PDA, a television with the necessary logic, etc. One difficulty with 
standard access methods is that with the increased improvement in video/audio 
compression technologies, access module logic may have to be updated frequently when 
new data formats are used. Further, for media presentations including different types of 
data, an access device (such as a general purpose computer or PDA) has sometimes to be 
loaded with several different access modules to access different media. 

SMIL and SMIL P layers and Authoring Took 

To facilitate the creation and distribution of multimedia content on the web, in 
June of 1998, the World Wide Web Consortium (W3C) announced the first specification 
of the Synchronized Multimedia Integrated Language (SMIL). This language provides 
some mechanisms for structuring various elements of multimedia content, including 
audio, text, image, and motion video and provides a way for web authors to schedule the 
playing of time-based media within HTML documents. 

A SMIL document according to the specification is actually an ASCII text file, 
which authors can edit with almost any text editor. SMIL documents generally resemble 
HTML files and references included components from external URL locations. SMIL 
uses XML to provide the mechanism to define SMIL markup tags for associating timing 
and positioning relationships between multimedia objects, so that audio, videos, images, 
and text can be synchronized. 

SMIL enables web authors to position (location, size, z-index ordering) visual 
media objects and assign temporal attributes (begin time, duration or end time) among 
one and another. SMIL provides capabilities to create interactive (via hyperlinks) 
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multimedia presentations similar to those on computer-based interactive CDs. SMIL 
media objects (addressable by a URL) can either reside locally or be distributed over the 
web. SMIL media objects can be updated independently and remotely. 

A SMIL Player is used to schedule multimedia presentation in a SMIL document, 
5 retrieving media objects on the web using URLs (Universal Resource Locator) described 
in the document, parsing the timing relationship among downloaded media, and 
consequently displaying the presentation based on the correct timeline. The Java SMIL 
Player is a Java applet-based player that follows a platform-neutral programming 
paradigm and makes use of the XML Parser and Java Media Framework (JMF). It can run 
10 within any popular browser without the need to download any plug-ins or ActiveX 
controls. 

A number of SMIL authoring tools have been proposed, which can generally be 
grouped into two categories: general purpose tools designed to create synchronized 
multimedia content for any SMIL-compliant players; and special purpose tools designed 

15 to create synchronized multimedia content for particular players. While these authoring 
tools are powerful, they still require proficiency in SMIL programming, which limits the 
range of users. Currently most SMIL authoring tools in the market have several properties 
in common: they include a text-based editor; they include a WYSIWYG (What You See 
Is What You Get) layout editor; and they do not include a media timeline design wizard. 

20 T.A.G. SMIL Editor 1.0, for example, is a general purpose SMIL authoring tool, which is 
designed to create synchronized multimedia content for any SMIL-compliant players. 
SuperTool is a specific purpose SMIL authoring tool for Real G2 Player only. SuperTool 
has a point and click WYSIWYG layout interface. Developers can add available media 
types, arrange media's layout and sequence how they are played in SMIL composition. 

25 Prior Publications 

The following publications may be related to the invention or provide background 

information. Listing of these references here should not be taken to indicate that any 
formal search has been completed or that any of these references constitute prior art. 

Blakowski, G, and Steinmetz, R. A Media Synchronization Survey: Reference 
30 Model, Specification, and Case Studies, IEEE Journal on Selected Areas in 
Communications, Vol. 14, No. 1, pp. 5-35, Jan. 1996. 
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Synchronized Multimedia Integration Language (SMIL) 1.0 Specification, WWW 
Consortium, Jun. 1998,' available at www.w3.org/TR/REC-smil. 

SUMMARY 

The approach of the present invention is different from aforementioned works in 
part because a Media Missile is open to and can carry different media specification's 
requirements. In one embodiment, not only actions and data embedded in the media 
objects are part of the Media Missile, the Media Missile can also carry executable binary, 
signature, and auxiliary information (which, in specific embodiments, may be pointed to 
by URLs). Unlike the above mentioned prior streaming techniques, which do not 
describe carrying various active media content, an executable binary discussed below can 
be carried in a Media Missile to make it ready for a runtime loader on a presentation 
device to run the media missile executable without need for other preinstalled software. 
Thus, in specific embodiments, Media Missile makes uses of operating system's launch 
sequence to run an included executable component and is self-contained for both of the 
software components and media data. 

Media Missile, according to specific embodiments of the present invention, can 
benefit from technological advances such as Java or COM, but does not rely on them. 

In specific embodiments, the invention involves a method and/or system for 
providing streaming content that can be activated from a single resource locator and in 
response to an activation, begin transferring a streaming presentation in a multimedia 
container, wherein said container has the capability of including one or more executable 
code components, one or more streaming content components, and one or more sets of 
presentation data. A Media Missile launcher in further embodiments may receive from a 
user device data indicating available hardware and software components at said user 
device and this data may be used by a scheduler/parser at the user device to request or not 
request different components from the media missile repository. Therefore, a media 
missile can be transmitted with some executable code or no executable code, depending 
on which components are present at the user site. 

In a further embodiment, presenting a media missile according to the invention is 
self-directed and requires minimal active management by or presence of preinstalled logic 
components at a user device. 
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In farther specific embodiments, the invention involves a method and/or system 
allowing a creator (or author) to specify a streaming presentation including receiving 
indications from an author specifying one or more items of streaming content and 
specifying presentation parameters. The content and parameters may be located at a 
5 server remote from the author. 

The invention and various specific aspects and embodiments will be better 
understood with reference to the following drawings and detailed descriptions. In 
different figures, similarly numbered items are intended to represent similar functions 
within the scope of the teachings provided herein. In some of the drawings and detailed 

10 descriptions below, the present invention is described in terms of the important 
independent embodiment of delivering visual and/or audio content. This should not be 
taken to limit the invention, which, using the teachings provided herein, can be applied to 
other streaming data content. For purposes of clarity, this discussion refers to devices, 
methods, and concepts in terms of specific examples. However, the invention and aspects 

15 thereof may have applications to a variety of types of devices and systems. It is therefore 
intended that the invention not be limited except as provided by the attached claims and 
equivalents. 

Furthermore, it is well known in the art that logic or digital systems and/or 
methods can include a variety of different components and different functions in a 

20 modular fashion. The following will be apparent to those of skill in the art from the 
teachings provided herein. Different embodiments of the present invention can include 
different combinations of elements and/or functions. Different embodiments of the 
present invention can include actions or steps performed in a different order than 
described in any specific example herein. Different embodiments of the present invention 

25 can include groupings of parts or components into larger parts or components different 
than described in any specific example herein. For puiposes of clarity, the invention is 
described in terms of systems that include many different innovative components and 
innovative combinations of innovative components and known components. No 
inference should be taken to limit the invention to combinations containing all of the 

30 innovative components listed in any illustrative embodiment in this specification. The 
functional aspects of the invention, as will be understood from the teachings herein, may 
be implemented or accomplished using any appropriate implementation environment or 
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programming language, such as C++, Cobol, Pascal, Java, Java-script, etc. All 
publications, patents, and patent applications cited herein are hereby incorporated by 
reference in their entirety for all purposes. 

The invention therefore in specific aspects provides a streaming video/audio 
signal that can be played on various types of video-capable terminal devices operating 
under any types of operating systems regardless what type of players are preinstalled in 
the terminal devices 

BRIEF DESCRIPTION OF THE DRAWINGS 
FIG. 1 is a block diagram showing an overall architecture of a system showing 
both authoring and playback according to various embodiments of the invention including 
multiple novel elements. 

FIG. 2 illustrates an exemplary configuration of a network in which the present 
invention may be practiced. 

FIG. 3 illustrates an example media missile container format according to specific 
embodiments of the invention. 

FIG. 4 is a flow chart illustrating functional aspects of a multimedia player 
according to a specific embodiment of the invention. 

FIG. 5 is a flow chart illustrating an example method for accessing a media 
missile according to a specific embodiment of the invention. 

FIG. 6 illustrates the operation of different timers initiated by a scheduler 
according to specific embodiments of the invention. 

FIG. 7 shows a standard browser display of example source files that may be 
packaged into a media missile in an example in accordance with specific embodiments of 
the present invention. 

FIG. 8 shows an example interface for packaging source data into a media missile 
according to specific embodiments of the present invention. 

FIG. 9 illustrates an example general metadata only media missile format 
according to specific embodiments of the invention. 

FIG. 10 illustrates an example metadata only media missile data structure 
according to specific embodiments of the invention. 
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FIG. 1 1 illustrates an example media missile data structure according to specific 
embodiments of the invention. 

FIG. 12 illustrates an example binary encoded metadata only media missile data 
according to specific embodiments of the invention. 

FIG. 13 illustrates an example presentation of a media missile in a player 
according to specific embodiments of the present invention. 

FIG. 14 illustrates an example general method for guiding a user in creating a 
media missile or presentation data according to specific embodiments of the invention. 

FIG. 15 illustrates an example of a Synchronized Authoring Manager (SAM), also 
referred to as a Synchronized Authoring Tool (SAT), according to specific embodiments 
of the invention. 

FIG. 16 illustrates an example of a Preview Manager GUI according to specific 
embodiments of the present invention. 

FIG. 17A-C illustrate examples of choosing various presentation templates 
according to specific embodiments of the present invention. 

FIG. 18A-B illustrate adding and previewing media according to a Video Lecture 
Template according to specific embodiments of the present invention. 

FIG. 19A-B illustrate adding and previewing media according to a Headline News 
Template according to specific embodiments of the present invention. 

FIG. 20A-B illustrate examples of creating presentation specification metadata 
according to specific embodiments of the present invention. 

FIG. 21 illustrates an example of a Document Generation GUI according to 
specific embodiments of the present invention. 

FIG. 22 is a block diagram showing a representative example logic device in 
which aspects of the present invention may be embodied. 

DESCRIPTION OF SPECIFIC EMBODIMENTS I 

In specific embodiments, the present invention involves a streaming media 

structure suitable for providing multimedia streaming over a communication data network 
including a cable network, a local area network, a network of other private networks and 
the Internet. The detailed description of the present invention include numerous specific 
details that are set forth in order to provide a thorough understanding of the present 
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invention. However, it will become obvious to those skilled in the art that the present 
invention may be practiced without these specific details. In other instances, well known 
methods, procedures, components, and circuitry have not been described in detail to avoid 
unnecessarily obscuring aspects of the present invention. 

The present invention is presented largely in terms of procedures, steps, logic 
blocks, processing, and other symbolic representations that resemble data processing 
devices. These process descriptions and representations are the means used by those 
experienced or skilled in the art to most effectively convey the substance of their work to 
others skilled in the art. The method along with the system to be described in detail below 
is a self-consistent sequence of processes or steps leading to a desired result. These steps 
or processes are those requiring physical manipulations of physical quantities. Usually, 
though not necessarily, these quantities may take the form of electrical signals capable of 
being stored, transferred, combined, compared, displayed and otherwise manipulated in a 
computer system or electronic computing devices. It proves convenient at times, 
principally for reasons of common usage, to refer to these signals as bits, values, 
elements, symbols, operations, messages, terms, numbers, or the like. It should be borne 
in mind that all of these similar terms are to be associated with the appropriate physical 
quantities and are merely convenient labels applied to these quantities. 

According to one aspect of the present invention, a streaming media structure is 
described below. The streaming media structure is provided to create, deliver, 
reassemble, and render synchronized multimedia content with its lightweight machine- 
aware components through streaming services. Each stream represents a user access to a 
video/audio title provided in a host from a terminal device. 

FIG. 2 illustrates an exemplary computer communication configuration in which 
the present invention may be practiced. Central server 102 together with a video/audio 
database 104 is a video/audio source comprising video/audio files that can be accessed on 
demand. As used herein, video/audio files or titles are referred to any video footage, 
video films and/or video/audio clips that can be compressed or packed in any format. One 
of the exemplary formats is Moving Picture Experts Group (MPEG). To play MPEG 
video files, one needs need an MPEG viewer or a client software executing in a personal 
computer with sufficient processor speed, internal memory. Another one of the popular 
formats for audio is MPEG-1 Audio Layer-3 (MP3), a standard technology and format for 
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compressing a sound sequence into a very small file (about one-twelfth the size of the 
original file) while preserving the original level of sound quality when it is played 
through a client program (player) downloadable from many web sites. It should be noted, 
however, that the exact formats of the video files do not affect the operations of the 
5 present invention. As will be noted and appreciated, the present invention applies to any 
formats of the video or audio files as well as other multimedia data that may include, but 
not be limited to, binary data and files, hypertext files and scripts. 

Preferably, data network 106 is a data network backbone, namely a larger 
transmission line. At the local level, a backbone is a line or set of lines that local area 

10 networks connect to for a wide area network connection or within a local area network to 
span distances efficiently (for example, between buildings). On the Internet or other wide 
area network, a backbone is a set of paths that local or regional networks connect to for 
long-distance interconnection. Coupled to data network A 106, there are two 
representative proxy servers 108 and 110 that each service representative terminal 

15 devices 1 16-1 19 via data network 1 12 and 1 14 respectively. It should be noted server 102 
sometimes named "central" or "central video" server does not necessary mean that server 
102 is only served as a repository of all the video titles. In some case, server 102 may 
service terminal devices directly and may retrieve other video/audio files from other 
servers such as proxy servers 108 and 1 10. 

20 Data network 112 and 114 are typically the Internet, a local area network or 

phone/cable network through which terminal devices can receive video files. The 
terminal devices may include, but not be limited to, multimedia computers (e.g. 116 and 
119), networked television sets or other video/audio players (e.g. 117 and 118). Typically 
the terminal devices are equipped with applications or capabilities to execute and display 

25 received video files. For example, one of the popular applications is an MPEG player 
provided in WINDOWS 98 from Microsoft. When an MPEG video file is received in 
streaming from one of the proxy servers, by executing the MPEG player in a multimedia 
computer, the video file can be displayed on a display screen of the computer. 

According to one aspect of the present invention, the streaming media structure is 

30 provided to create, deliver, reassemble, and render synchronized multimedia content with 
its lightweight machine-aware components through streaming services. 
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Cases 1 and 2 below present detailed procedures of how to construct the streaming 
media structure relying on a complied and linked version implementing the present 
invention and executing preferably in a client device. As such a video/audio title can be 
submitted to a host device in a format compliant to the streaming media structure. Cases 
3 and 4 below present detailed procedures of how to playback video/audio titles packed in 
a streaming media data structure according to embodiments of the present invention. 

It can be appreciated by those skilled in the art that one of the important features 
in the present invention is a new versatile streaming data file format or structure that can 
be accessed from any types of terminal devices operating under any types of operation 
systems. Another important feature is that the present invention, in specific embodiments, 
allows the elimination of preinstalled players. A streaming media structure according to 
specific aspects of the invention can include a built-in player that is downloaded on the 
fly and then operates in a local machine to access the streaming data and to find and 
install any necessary additional executable components. As will be understood to those of 
skill in the art from the teachings herein, elimination of preinstalled players for various 
types of streaming content allows streaming content to be more easily accessed on 
devices with limited code storage, such as cell-phones or other hand-held or versatile 
playback devices. 

The processes, sequences or steps and features discussed herein are related to each 
other and each are believed independently novel in the art. The disclosed processes and 
sequences may be performed alone or in any combination to provide a novel and 
nonobvious file structure system suitable for media delivery system. It should be 
understood that the processes and sequences in combination yield an equally 
independently novel combination as well, even if combined in their broadest sense. 

The streaming media structure as described herein, in accordance with one aspect 
of the present invention is robust, operationally efficient and cost-effective. The 
mechanism provides the best use of servers providing streaming services and permits 
seamless delivery of streaming video with highest quality of services possible. In 
addition, the present invention may be used in connection with presentations of any type, 
including sales presentations and product/service promotion, which provides the video 
service providers additional revenue resources. 
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In order to facilitate description, the following discussion will describe the present 
invention in terms of streaming media presentations. It will be understood to those of skill 
in the art, however, that the invention may be used in other streaming data situations. The 
invention should therefore not be taken as limited except as provided in the attached 
5 claims. 

In various embodiments, systems or methods according to various aspects of the 
present invention, can create, deliver, reassemble, and render synchronized streaming 
multimedia content with lightweight machine-aware components through streaming 
services. Each stream represents a user access from a client on one computer to a server 

10 on a different computer. As will be understood from the teachings herein, a system or 
method according to the present invention can handle various types of streaming data 
including real-time multimedia content as part of a presentation or multimedia missile 
through streaming services. 

In particular embodiments, an author (a user or a program) creates synchronized 

15 presentation using a wizard-like Synchronized Authoring Manager (SAM). In further 
embodiments, an SAM can include lightweight software components and/or indications 
of such components with the media data content into a Media Missile container that is 
streamable from a single location. 

The present invention, in some embodiments, thus allows that the streaming of 

20 active and passive content can be done all at once and can be self-contained and 
guidance-free. The active content is the various machine-aware runtime components. The 
passive content is the media data to be rendered at runtime. All of the presentation 
content (both active and passive) can be stored in a container-like file format or object- 
relational database. A special header can be attached to individual media objects to 

25 specify metadata information such as signature, data length (or offset), and linkage. A 
Presentation Specification can be generated in descriptive tagged language such as 
Synchronized Multimedia Integration Language (SMIL) from World-Wide Web 
Consortium (W3C). 

A container-like file format of a Media Missile has advantages for content 
30 providers in the convenience and portability of the content. With the flexible file 
specification of the Media Missile, a content provider of a synchronized Multimedia 
presentation can create the presentation and then store, manage, distribute, or serve the 
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presentation as easily as they would a single file. Thus, a synchronized multimedia 
presentation, using a media missile, can be as easy to deliver or manage as a standard GIF 
or JPEG banner ad. Conventional SMIL-based or HTML-based presentations cannot 
merge multiple objects into a single container and can require managing and downloading 
dozens of individual files to render properly. 

A media player implementation, such as a an SMIL player, that is compliant to 
this design can render and playback media contents in real-time in accordance with a 
presentation specification. In particular, individual media content objects, specified in the 
presentation, corresponds to object information in the metadata and media data in the 
media missile. 

In a further embodiment, a one-way one-to-one hashing function or other method 
can be used to generate a signature for the media. The signature can be both a watermark 
of the media, and also be a search key (to be used in place of a name) if the media is 
stored in a large distributed media repository. In a further important embodiment, a media 
missile format enables synchronized presentation on mobile platform with limited pre- 
installed software capability or memory footprint. 

FIG. 1 is a block diagram showing an overall architecture of a system showing 
both authoring and playback according to various embodiments of the invention including 
multiple novel elements. Various aspects and details of the designs are elaborated in later 
sections. The overall architecture shown in FIG. 1 includes the following functional 
and/or structural elements. 

Media Missile (MM) 12 is a container specification that allows inclusion of 
various items of machine-aware executable binary (or script), media data, and associated 
information such as presentation specification or metadata. (MM) can be a container of 
container(s) in a nested structure. The machine-aware executable binary (or script) is can 
also be referred to as active media content. The active media content can be implemented 
in machine-neutral programming paradigm (such as Java) or machine-dependent code 
(such as .exe files). Media Missile with active media content is self-contained and 
guidance-free for synchronized real-time multimedia authoring and/or presentation over 
network environment. Once the media missile is indicated and the active media content is 
retrieved and loaded to a machine, it is ready to run by itself. 
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Synchronized Authoring Manager (SAM) 10 takes and understands various 
language specifications, formats, and templates in synchronized multimedia, including 
Media Missile. Language specification and templates such as SMIL-compliant ones are 
used as input to SAM's authoring wizard. In specific embodiments, an SAM wizard 
5 guides a user to compose a synchronized presentation and retrieve media data from Media 
Repository Manager 20. In one option, an end result of an SAM is a packaged Media 
Missile. As an option, an SAM 10 can also output a separate presentation specification 
(such as an SMIL presentation specification) that is compliant to international or industry 
standards. 

10 As shown in FIG. 1, SAM 10 can receive as inputs templates, media missile 

specification, multimedia language specification, and media data, which may be received 
from a media repository. Templates, as understood in the art, include data describing a 
general format, layout, or structure for a presentation. SAM 10 creates a media missile 
and/or presentation specification, which is stored in repository 20. 

15 Media Repository Manager (MRM) 20 handles storage and retrieval of real- 

time (streaming) and non-real-time data accessible by a client by activating (or 
"shooting") a media missile. The real-time data (or clocked media data) 22, in a specific 
example, can refer to media data or media missiles stored or broadcast from Streaming21 ! s 
Media Server and Caster or other compatible video server platforms, which are 

20 collectively illustrated as data source 30. The non-real-time data is the presentation 
specification or any other auxiliary information stored on a web server or file server. In 
the figure, Data Store 40 illustrates generally one or more stores for additional digital 
data. 

As shown in FIG. 1, MRM 20 can output either directly to an SMP 50 or through 
25 a web or other front-end interface 55. Also, as shown, MRM 20 acts as a middle manager 
for communicating media data with SAM 10 and managing data from source 30 and data 
store 40. 

(MRM) 20 also can be understood as representing access to any media data that is 
available to SAM 10 and SMP 50, such as any authorized and available computer on the 
30 Internet. 

Synchronized Multimedia Player (SMP) 50 has an internal scheduler to process 
delay-sensitive data. It can also handle sequencing of static media data such as images 
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and text. It complies with the SMIL recommended by W3C. Hence, it is able to interact 
with web servers to retrieve and/or to utilize SMIL presentation specifications. In 
addition, an SMP can also work with a Media Missile locally on a single platform or in a 
distributed network environment. 

As shown in FIG. 1, SMP 50 can receive data directly from MRM 30, or through 
interface 55. An SMP 50 can also be directed to receive media data directly from a data 
store 40. 

Media Missile 

FIG. 3 illustrates an example media missile container format according to specific 
embodiments of the invention. This example shows a very specific example bit-wise 
arrangement of various elements. It will be understood to those of skill in the art from the 
teachings herein that variations of the field and bit arrangements and formats exist within 
the scope of the invention. Particularly, logic true may be encoded as a variety of 
different bit values and field lengths may vary, included using more than one bit for fields 
indicated as bit fields below. The container format shown in FIG. 3 is further described 
below. 

Media Missile 12 is a document container, which can include active content and 
passive media content. The active content is machine-aware executable binary or script. 
Passive content can include a presentation specification, metadata information, and the 
media data. The media data can be another Media Missile. Thus, Media Missile has a 
nested structural format. All of the content of a media missile can be stored in a file, 
directory structure, or database, such as on a server system or computer such as repository 
303. In a further embodiment, according to specific implementations, some or all 
components can be generated on the fly by a multimedia missile delivery system. 

The executable binary (or script) 200 and the presentation specification 210 are 
both optional. If machine-executable code is not present, E-Length 205 is zero. The 
Continuation Bit 202 tells what to expect for the follow-on Media Missile structure. If 
Continuation Bit 202 is one and E-Length 205 is zero, this indicates that the entire Media 
Missile starts from a presentation specification (210). If Continuation Bit 202 is zero and 
E-Length 205 is zero, it indicates that the entire Media Missile starts from MetaData 
information 220. Finally, if Continuation Bit 202 is zero and E-Length is "-P, the entire 
Media Missile consists of pure media data. 
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Continuation Bit 212 in front of P-Length 215 tells whether the metadata 
information is included in the media missile (1) or not (0). If no metadata information is 
available, the data right after Presentation Specification is the media data. 

If metadata information 220 is present, the first bit 222 tells whether the data right 
5 after the metadata information is another Media Missile (1) or just pure media data (0). 
In metadata information, the signature field 226 is 128-bits by default to store a digital 
signature using one-way one-to-one hashing function such as MD4 (Message Digest 4) or 
MD5 (Message Digest 5). The length of the signature field can also be implicit by the 
implementation to guard against trivial breakdown and guessing. 

10 Following signature field 226, MetaData 220 holds one or more length field(s) 

(such as 245 and 295) for media data. Each length field is 64-bit starting with a bit 
representing URL (1) or payload (0) (such as 242 and 292). In specific embodiments, 
each length field in MetaData 220 corresponds to a segment of data in media data 230. 
The actual data stored in 230 can be characters indicating a URL, or can be the media 

15 data. Segment(s) of data are stored in sequence according to the sequence of length 
field(s). Here, URL stands for Universal Resource Locator for World-Wide Web 
hyperlink; payload indicates media data for the media player. As will be understood in 
the art, however, URL generally can refer to a locator (such as a filename or database id) 
for any resource available via a computer or communications system. In specific 

20 embodiments, a URL or other locator can indicating streaming or "real-time" data 
sources, such as sources of live streaming data. 

As will be understood from this discussion and other descriptions herein, an 
important and innovative aspect of the Media Missile Container Specification is that once 
a user "shoots" the Media Missile from a particular location, from the perspective of the 

25 user, the contents of the Media Missile can be delivered apparently from a single location. 
In contrast, with formats such as a standard HTML or SMIL type page, a client at a user 
site may have to access dozens of different text, static image, executable, or audio and 
video files, from dozens of different locations, to complete a presentation. With a Media 
Missile, a client can access all of the included data from a single location, with the data 

30 delivered in the Media Missile format. This aspect also has advantages for data providers 
in that Media Missile formatted data is easier to manage, distribute, and deliver than is a 
presentation the requires many different files. 
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Examples of Authoring Media Missile 

What follows are descriptions of the operation of authoring manager 10 according 

to specific embodiments of the present invention. A wizard, as discussed below, is 
understood in the art as logic implemented processes that guides a user through a 
sequence of steps in an assisted and constrained fashion to accomplish a result, such as 
installing software or creating a particular document or file type. A template, as discussed 
below, is understood in the art as set of data specifying general format parameters that 
may be used to determine or guide the final format of one or more final files or data 
structures. Synchronization primitives are further described below and illustrated in FIG. 
6. 

Example Basic Authoring Method 

The steps to create or update a Media Missile (using Synchronized Authoring 
Manager (SAM) 10 are as follows. 

L The Synchronized Authoring Manager (SAM) provides template selection for 
user to create a synchronized multimedia presentation. It can also read from an existing 
presentation specification for user to update the specification. 

2. SAM takes presentation specification and media data as input used to parse and 
create a tree representation internally. The tree is a top-down representation of the 
synchronization primitives. Each intermediate node is either a parallel or sequential 
synchronization primitives. SAM then retrieves or calculates the required fields according 
to the Media Missile specification for the leaf nodes (individual media) in the tree. This 
can be a pipeline process. 

3. SAM attaches system-required fields, e.g., a 64-bit trailer, to individual media 
specified in the presentation to become a basic Media Missile with media data only. 

4. SAM uses a depth-first tree traversal to calculate and create optional fields 
based on user input. The optional fields are feasible to apply to any node in the tree 
depending on user's needs. 

5. SAM uses a breadth-first tree traversal to traverse the entire tree and create the 
final Media Missile. 
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Sample Sequences of Authoring Interactions 

Case J: Creating Synchronized Multimedia Presentation for a Guided Lecture: 

1 . A wizard helps author to pick up a lecture template or other template of the 

presentation skeleton through a point-and-click list, where corresponding layout of the 
5 template also displays. 

2. The wizard asks the author to pick up a lecture video and corresponding 
presentation slides in sequence. 

3. The author has a choice to use default duration for slide display duration or use 
MetaData Manager to customize the presentation timeline. 

10 4. If using the MetaData Manager, the wizard displays both the video and 

corresponding slides at a particular point in time. Author has the liberty to modify the 
duration setting for each slide. 

5. The wizard then prepares the final output, both the presentation specification 
and media object, in user-specified formats, e.g., SMIL for presentation specification or 

15 Media Missile for media object. 

6. The wizard asks author whether author wishes to keep the final output locally 
or upload it to a specific hosting server. 

Case 2: Updating Synchronized Multimedia Presentation for a Guided Lecture: 

1. Author uses wizard to pick up an existing presentation specification or media 

20 missile from a hosting server. 

2. The wizard asks the author to delete or add video and slides. 

3. The author has a choice to use default duration for slide display duration or use 
MetaData Manager to customize the presentation timeline similar to step 3 in Case 1 . 

4. The wizard repeats step 4 and 5 in Case 1. 

25 5. The wizard asks the author to apply the changes and to check if object deleted 

from the specification should also be permanently removed from the hosting server(s). 
6. The wizard repeats step 6 in Case 1. 

Example Basic Method for Playing Back Media Missile 

Playback of Media Missile according to specific embodiments of the present 

30 invention involves three potentially overlapping processes: delivering, assembling, and 
rendering Media Missiles and their contents. 



-18- 



BNSDOCID: <WO 0126378A1_I_> 



WO 01/26378 



PCT/US00/27634 



Example Method for delivering a Media Missile are as follows: 

1. User can use a browser to point-and-shoot the Media Missile. Alternatively, 

user can also use command line interface to launch a local self-contained Media Missile* 
or run a player with a Media Missile name as input. 

2. If the Media Missile contains a complete executable binary (or script), it 
generally should be able to load itself and run (e.g. it can be loaded by the native 
operating system of the target device). In specific embodiments, Media Missile takes 
advantage of the runtime loader characteristics available on most operating system 
platforms such as UNIX or Windows. For example, runtime executables or scripts are 
identified by a Magic Number (first two bytes in any file) on UNIX platforms. On 
Windows platforms, runtime executables and scripts are explicitly identified by a three- 
letter filename extension registered in the system registry (such as .EXE or .COM). A 
loader generally also checks to determine if the files are indeed executable. Other 
operating system platforms may have different mechanisms to load and launch runtime 
executables that may be used by a media missile. 

3. The Media Missile will be streamed and downloaded from an application server 
or a web server if it is at remote site. A browser for a web or application server is 
generally able to run or save a remote file. To make Media Missile compatible with these 
servers, a hyperlink retrieval of a Media Missile in a web-enabled page will implicitly 
stream only the machine-aware portion of the Media Missile. The machine-aware portion 
can then be launched and run on a local platform and manage requests for additional 
portions of the media missile. Because the Media Missile's executable footprint size is 
known ahead of time, at runtime the executable is able to manipulate the rest of the Media 
Missile according to its specification. 

Example Method for presenting a Media Missile are as follows: 

1. The Synchronized Multimedia Player (SMP) can be locally-resident executable 

code or executable code that is downloaded or streamed with the Media Missile. SMP has 
a timeline-based scheduler to schedule the playback according to the presentation 
specification. The presentation specification again may be available within the Media 
Missile or from another source, such as through web page that is holding the Media 
Missile. 

2. SMP parses the presentation specification to build a presentation tree. 
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3. SMP fires the presentation from the top of the tree based on the synchronization 
characteristics (either parallel or sequential) of each intermediate tree node. For parallel 
nodes, SMP creates independent execution threads to handle each parallel task and each 
of these threads refers to a global time base. For sequential nodes, SMP can use a single 

5 execution thread to step through the sequential tasks. 

4. When SMP fires a leaf node, SMP locates and launches the media player and 
media data. 

If the link to the media data in the specification is a relative path, SMP will by 
default search the Media Missile itself, using the media data's signature or other identifier 

10 to see if the media data is available in the Media Missile. This searching can take place on 
a Media Missile either that exists already at a local machine or on a Media Missile still 
resident at a remote server. Various protocols are known for a streaming data client to 
make requests and receive responses from a streaming server and the protocols can be 
used to search for specific data within the Media Missile. Otherwise, SMP will follow an 

15 . absolute path either specified in the Media Missile or from the root of the server where 
the Media Missile or presentation specification was streamed, to locate the media data. 

Regarding the player, SMP will look to determine if the player for the particular 
media data is located on its local machine. If the player is not located on its local 
machine, SMP will search for the player in the Media Missile or in other data store. 

20 Playing a Media Missile 

FIG. 4 illustrates generally the functional elements the enable MM playback 

according to specific embodiments of the invention. As shown in the figure, MM 
playback may access media and code data from a variety of locations. A media missile 
repository 303 can be understood as a primary data storage of media missile data, 

25 including passive data and active code data. The repository in specific implementations 
can be understood to reside on a server, accessible via a local network, wide area 
network, or the Internet. When residing on a server (as opposed to locally stored), 
repository can be understood to have capabilities of streaming session servers, including 
seek to end, search, play and pause capabilities in response to requests from a client. In 

30 some embodiments, the repository will generally store active and passive content needed 
to play a media missile on a variety of platforms, though only content needed for a 
particular platform may be downloaded from 303 by an SMP. 
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Local Code Resource 305 can be understood as those code components available 
at the local device. For some devices, such as a computer workstation on a LAN, the 
local code resource 305 may be quite extensive. For such devices, a MM scheduler and a 
variety of players may be permanently or nearly permanently present at the local device 
and therefore not have to be transferred from a server to play a media missile. For other 
devices, such as a cellular telephone for example, there might be very little or no local 
code resource and all code will need to be downloaded from 303. 

Web or Application Server Repository 307 can be understood as storage for any 
active or passive data not in 303 or 305. As described herein, this data may include data 
referenced by a resource locator (such as a URL) within a Media Missile. An example of 
such data would be an HTML page or data used to complete a presentation, advertising 
banners or data from external sources, or any other data referenced in the medial missile 
or needed to complete playback. 

As elsewhere described herein, to begin playing a media missile, an executable 
binary or script component able to parse a MM is first initiated on a playback device. 
This executable, as described herein, may be initially loaded from 303, 305 or 307. An 
initial task, in particular instances, is to parse a presentation specification 402, if present, 
and then to locate and execute a Real-Time Presentation Scheduler 404. Scheduler 404 
then schedules the downloading (if necessary) and initializing of any players 406 needed 
to playback the actual media data. These players can include a variety of off-the-shelf 
media players, such as QuickTime™ or RealMedia™ players or the players can include 
proprietary players. As indicated in the figure, once scheduler 404 has initialized, it 
schedules the downloading and initialization of any players. Thus, the downloading and 
initializing of various player and player components is under control of the overall 
presentation schedule being run by the scheduler. 

As shown in the figure, scheduler 404 also determines the ending of a presentation 
and terminates the necessary components. 

The steps to render or playback a Media Missile are as follows: 
1. Software (SW) checks if the Media Missile contains an initial executable code 
portion (binary or script) for the platform. If the checking is positive, go to Step 2. If the 
checking is negative, goto Step 3. This checking can be accomplished by software 
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registered at the target device to handle a MM file type or the checking can be self 
starting if the MM itself is an executable file or script. 

2. Since there are machine-aware components in the Media Missile, it can be 
launched on the fly, or it may be used as an enhancement of the SW already present of the 

5 system to render the media data within the Missile. 

3. In case there's no machine-aware executable within the Media Missile, the 
media missile or Operating System will trigger and launch a local SMP with default 
components to read media data from the Media Missile. 

4. SMP handles both barrier and event synchronization with elastic timers for 
10 synchronization point. 

SMP internally has to handle synchronization events according to the presentation 
specification 320. Certain specifications define precise timing sequence. Some others 
have relaxed schedule. Any playback of media(s).can be either sequential or concurrent 
(parallel). 

15 Sample Sequences of Playback Rendering 

Case 3: Playing Back a Synchronized Multimedia Presentation for objects in Media 
Missile format in a Web Page: 

1. Client (SMP) 50 accesses a synchronized multimedia presentation, e.g., 
formatted in a HTML/XML web page. 
20 2. The names/signatures of the media objects at particular points in time in the 

presentation-timeline are passed to the player(s) in the web page through a player 
scheduler which is downloaded on the fly or resides on the local computing device. 

3. Each player starts to retrieve a media missile header corresponding to the media 
object name/signature. 

25 4. Each player inspects the header to verify the necessity of downloading and 

registering embedded lightweight software components in the media missile to render the 
media data. 

5. Each player realizes and loads media-specific rendering components at runtime. 

6. For a parallel start of rendering, the scheduler sets a barrier synchronization 
30 primitive, further description of which is provided below. 

7. Each player is enabled to play back and render media data according to the 
"Enable" signal from the player scheduler, the starting time, and the duration in the 
presentation specification. 
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?? e .f : f layin S Back a Syntonized Multimedia Presentation for objects in Media 
Missile format in a distributed environment: 

1. A self-contained guidance-free Media Missile, which is downloaded or stored 
in local storage, is launched. The beginning of the Media Missile is the player scheduler, 
which can be a script or a binary executable object compatible with the runtime platform. 
Hence, it would run by itself. 

2. The player scheduler then loads a presentation specification, which is attached 
after any present runtime executable code segment in the Media Missile. 

3. The names/signatures of the media objects and their position in the Media 
Missile at a particular point in time in the presentation timeline are passed to the player(s) 
specified in the Media Missile through the player scheduler. The player(s) (which can 
also be a Media Missile by itself) resides in a distributed network environment. Hence, 
the name of a media object can be a link to a Media Missile over the networks. 

4. The procedure then repeats step 3 to 7 in Case 3. 

FIG. 5 is a flow chart illustrating an example method for accessing a media 
missile according to a specific embodiment of the invention. 

Playback Timing 

To handle both of the sequential or parallel playback of media contents with either 
precise or elastic timers, an SMP defines the following states: started, initialized, played, 
paused, and stopped for each media player under its control. FIG. 6 illustrates the use of 
these states with the synchronization primitives and the timing requirements. SMP 
attached a timer for each synchronization primitive to control the duration. Each media 
player may also have attached timers to control both delay and duration. Whether the 
content has additional embedded timing information or not, it can be played back as if it 
is a continuous media, i.e., media with timing information. Thus, SMP turns some static 
media types into time-based media such as text or image. The implementation of the 
synchronization checkpoints in the Real-Time Presentation Scheduler at any point in time 
is elaborated as follows and as illustrated in FIG. 6. 

Bl. Sequential Playback with Precise Timer: 

In this case, the scheduler 330 pairs the started and the stopped time of any two 
media players 340. As such, a player may be launched, based on estimated latency, to 
cache (or buffer) media data before it starts to render the data. 

B2. Parallel Playback with Precise Timer: 
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In this case, the scheduler checks for barrier synchronization on started, 
initialized, played, and stopped states for all of the players. Each player may be toggled 
between played and paused states if delay timers are set within the player. 

B3. Sequential Playback with Elastic Timer: 
5 In this case, the scheduler pairs the started and the stopped time of any two media 

players. However, one must be stopped before another is started. Alternatively, it can be a 
single media player that takes sequential input scheduled in time. 

B4. Parallel Playback with Elastic Timer: 

In this case, the scheduler only sets barrier synchronization for started and stopped 

10 states for all of the players. Each player controls its own status and the scheduler ignores 
any events from players other than started and stopped events. 

As will be understood to those of skill in the art from the teachings herein, the 
synchronization of multiple players allows the presentation of multi-part media 
presentation. Case B2, for example, might illustrate a case where a presenter or lecturer 

15 is narrating a video clip or a slide-show presentation. The presenter will come to a point 
in the presentation where it is time to begin the video clip, the clip will begin, and the 
presenter audio or audio and video, will continue while the video clip proceeds. The 
video clip might occupy an entire screen space, or the clip might occupy a subwindow in 
the screenspace, while video of the presenter occupies a different window portion. As 

20 shown in FIG. 6, initialization of a Player2 can begin at some point during the activation 
of Playerl. Thus, Player2 or Player3 components need not be downloaded early and 
might not be downloaded at all if a user halts ends access to a presentation. It will be 
understood from the teachings herein, as discussed with respect to FIG. 3, that the format 
of the media missile allows a multi-player presentation to continue from a media missile 

25 location, downloading lightweight software components as needed according to different 
timer strategies as shown in FIG. 4. 

Example Application (WebAds) 

Further understanding of aspects of the present invention will be provided by 

considering the following example. This specific example is provided to illustrate 
30 various aspects of the present invention, but the invention should not be limited by any of 
the details provided herein. This example involves use of the herein described media 
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missile format, according to specific embodiments of the present invention, to enable a 
WebAds streaming presentation. 

Using A Packaging Utility to Package a MM 

As a simple example, a MM can be created from an existing HTML presentation 
as described below. In a particular embodiment, a package utility may be launched as 
follows: 

package webAdsUrl [webAdsDir startingHtmFile] 
In this example, webAdsUrl indicates a URL pointing to an external HTML 
location, in case the remainder of the packaged file is corrupted or empty when the MM is 
accessed. webAdsDir indicates an absolute or relative path to specify a directory the 
includes files to be placed in the MM and startingHtmFile indicates an initial html file 
that will be used as the initial outline of an MM. As an example, consider a Web Ads 
directory that includes the following files: 
c:\webAds\s2 lWebAds.htm 
c:\webAds\otherLinks.htm 
c:\webAds\logo.gif 
c:\webAds\sponsor\abcCo.jpg 
In this example, the package utility can be run with a command line such as: 

c:\s21\bin\package "http://www.abcCo.com" c:\webAds s21 WebAds.htm 
This example can be further understood with reference to FIG. 7 to FIG. 13. FIG. 
7 shows a standard browser display of example source files that may be packaged into a 
media missile in an example in accordance with specific embodiments of the present 
invention. According to various specific embodiments of the present invention, and as 
will be understood from the teachings provided herein, this source data file could be any 
type of media data. Here, the example media data includes an HTML file and its 
included image files, including a GIF and a JPG file. The HTML file also includes a 
reference to another HTML file, otherLinks.htm, which could include any additional links 
or html-type data. 

FIG. 8 shows an example interface for packaging source data into a media missile 
according to specific embodiments of the present invention. A variety of different 
packaging tools can be used in different embodiments of the invention, including 
advanced wizards for creating MM as described herein. In this example, however, a 
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simple packaging tool is used to create WebAds, and this packaging tool requests just the 
four parameters illustrated. 

When playing such a file, a player can extract the packaged file and place it into a 
temp directory. If the file is an audio-only clip, a default HTML browser page can be 
displayed, or the starting html file (e.g. s21WebAds.htm, as shown in FIG. 13) can be 
displayed in the player window. When the player quits, it generally will remove the 
temporary files and temporary directory. 

If player.dat is missing, a player can use a centrally located file (such as 
www.Streaming21.com/ads) or show error and then stop the application. If player.dat is 
corrupted or player.dat contains no files, a player will use the default URL inside the 
player.dat, if possible. In the worst case, the player can display the default bitmap for 
Audio playback. 

FIG. 9 illustrates an example general metadata only media missile format 
according to specific embodiments of the invention. In this example, the file begins with 
metadata information. As indicated in the figure, the metadata includes one or more 
URLs for associated files with the streaming presentation. Following the metaData is the 
mediaDataBlock, which contains the actual media object data or locators for media object 
data. 

FIG. 10 illustrates an example metadata only media missile data structure 
according to specific embodiments of the invention. Note that in this example, the values 
shown at the top of the figure are assumed. Note that as indicated in the figure, all of the 
source files are incorporated into the media data area of the media missile file, with the 
metadata area indicating the identity of the portions of data in the media data area. 

FIG. 1 1 illustrates an example media missile data structure according to specific 
embodiments of the invention. Comparing this figure with FIG. 3 will show one specific 
data structure example of how a Media Missile according to specific embodiments of the 
present invention may be realized. Note that objects in 

FIG. 12 illustrates an example binary encoded metadata only media missile data 
according to specific embodiments of the invention. 

Optional processing for WebAd in a media missile allows ignoring an Executable 
Binary of Script portion and a Presentation Specification portion, if either are found in the 
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media missile as, for example, shown in FIG. 11. . An example of seeking in a media 

missile file to skip this data is as follows: 

OpenFile 

Seek to ending- 8bytes 

BOOL continationjDit = readlBitO; 

int64 eLength = readNextBits (63) ; 

SeekBackToBeginning ; 
if (eLength==-i) 

else Y/TeLength) " re ^ ired metaDataBlock is absent 

•p , s) ^P Firs . tB y tes(eLen 9th); // skip Executable Block 
if (p_blk_exist) 

{BOOL next_blk_is_meta_data = readlBitO- 
if s Onext_blk_is_meta_data) error;// required metaDataBlock is absent 

{ int64 p_blk_len = readNextBits (S3) ; 

}S kipFurtherBytes(p_blk_len); // skip Presentation Block 

) // Reach Meta Data 

FIG. 13 illustrates an example presentation of a media missile in a player 

according to specific embodiments of the present invention. In a further embodiment, a 
player can use a local ads.html page by default, instead of always needing to contact a 
server for the html page. The local ads.html can include references to other files, such as 
avi, gif, jpeg, html or similar files. In specific embodiments, these may be restricted to 
the same directory in which ads.html resides and its sub-directory. Ads.html and related 
files can be packaged into a media missile format, with, as an example, a name in the 
form 'player.dat'. 

EXAMPLE SYNCHRONIZED AUTHORING MANAGER IMPLEMENTATION 

An SAM according to specific embodiments of the invention provides a user- 
friendly GUI that allows an author to successfully complete a Media Missile (or, in some 
embodiments, just a well-formed Presentation Specification) by applying mouse clicks 
and keystrokes, with no programming background required. SAM provides a wizard- 
driven guide to help non-programming users generate multimedia presentations. SAM 
wizards generate MM based on a series of templates, so that the multimedia presentations 
created through SAM are well-formed and can be parsed and rendered by any compliant 
player. In further embodiments, SAM provides a tool to preview different media such as 
text, images, html files, audio and video clips and facilitates the generation of 
synchronization data between different media. SAM can be implemented in Java or other 
programming environments. FIG. 14 illustrates an example general method for guiding a 



-27- 



_0126378A1J_> 



WO 01/26378 



PCT/USOO/27634 



user in creating a media missile or presentation data according to specific embodiments of 
the invention. 

Template Support 

A SAM according to the present invention provides a set of different standard 
5 templates to allow authors to create presentations such as video lectures, headline news, 
and slide shows. 

For example, a video lecture template makes it easy for an author to generate a 
video lecture (video synchronized with HTML slides). 

A headline news template is designed for virtual briefings (video synchronized 
10 with text streams at bottom) in scenes such as: headline news - video with anchorperson 
and text containing headlines; online event — video with event introduction and text 
containing registration information; product advertisement — video with product 
advertisement and text containing pricing information. 

A slide show template is designed for illustrated audio (audio synchronized with 
15 images) in scenes such as: guided tour — narrated map with scenic photos to guide users 
through a site tour; process presentation — audio commentary to slides demonstration of a 
process; or online publication — sound book or magazine with reference materials in 
pictures. 

While some prior multimedia authoring tools supported only non-streamed data 
20 types, an SAM according to the present invention, allows inclusion of streaming media 
along with other media types. FIG. 15 illustrates an example of a Synchronized Authoring 
Manager (SAM), also referred to as a Synchronized Authoring Tool (SAT), according to 
specific embodiments of the invention. 

SAM Core Components 

25 In specific embodiments, SAM includes four components: Presentation Wizard, 

Preview Manager, Presentation Specification Data Manager, and Media Missile 
Generator. 

Presentation Wizard is SAM's kernel that provides an easy way to enhance 
disparate media into integrated and dynamic multimedia content. It guides users to create 
30 presentations in just a few steps. 

Preview Manager can display any allowed medium element. If the medium is 
time-irrelevant, such as an image, a text stream, or an html file, then the exact content 
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will be shown in the preview window. If the medium is time-intrinsic, such as an audio or 
video clip, then it will be played continuously from the beginning in the preview window. 
If the medium is from a live streaming source, such as a live camera or video feed, a live 
sample of the medium can be played in the preview window. The Preview Manager can 
be activated inside and outside the Presentation Wizard. Users can use the Preview 
Manager to preview the medium before incorporating it into a multimedia presentation. 

Presentation Specification Data Manager is responsible for creating 
Presentation Specification Data (referred to at times in this context as metadata, although 
herein it refers to data for the presentation specification and not the Media Missile 
MetaData described above) information and a presentation specification, which is used to 
synchronize different media within a multimedia presentation. Users can playback an 
audio or video clip or stream, meanwhile recording relevant timing information for static 
media based on the content of the clips. For example, a user wishing to create a 
presentation that displays a narrated map (using audio), showing some slides of scenic 
photos (using images), and access from a live outdoor camera, indicating current weather. 
With Presentation Specification Data Manager, the user can decide the timing point to 
show the next slide according to the audio content or to open a window to display the 
streaming (or live) data. The information is saved in a data structure and used later for 
creating the corresponding presentation content. 

Document Generator features automatic creation of a presentation specification 
and/or a Media Missile. The document's structure is based on one of the templates, and 
the media and timing information come from Presentation Wizard operation. The 
Document Generator can package the various media data with the presentation data to 
create and any necessary or executable content to create a Media Missile. As an option, 
the Document Generator can also output just an SMIL-type presentation specification. 
Creating a Media Missile Using SAM 
Step 1 - Choosing A Template 

At first, users choose the type of presentation they wish to create. Each template is 
designed with a specific purpose in mind. SAM can support templates such as: 
Video Lecture Template 

Video lecture usually consists of a video clip or a streaming feed from a streaming 
data source such as a live camera or continuously encoded television broadcast, 
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synchronized with HTML slides. The video can be any media format supported by a 
target Player. The HTML slides are any conforming HTML files or other files that can be 
handled by a player. In some embodiments, if the slides are originally in another format 
(such as Microsoft PowerPoint) they are first converted to HTML format using a "Save as 
5 HTML" command in PowerPoint. 

Headline News Template 

Headline news usually consists of a video clip or a streaming feed synchronized 
with text streams. The video clip can be any media format supported by a target player. 
Text streams are generally formatted ASCII text. In a specific embodiment, users can 
10 directly type any text they want in the second step. 
Slide Show Template 

Slide show usually consists of an audio clip or streaming audio feed synchronized 
with image slides. The audio clip can be any media format supported by a target Player. 
The image slides are image files in GIF, JPEG, or any supported format. 

15 Step 2 - Adding and Previewing Media (Preview Manager) 

The second step is to add media for the presentation. Users can activate the Open 

File dialog box to select the files or streaming sources they wish to add. Users can also 
activate the Preview Manager to preview any selected medium. In various situations, 
users can decide how to create presentation data by using either the Presentation 

20 Specification Data Manager or SAM's default setting. If the Presentation Specification 
- —-Data Manager-is chosen, -then- the wizard will-activate -the -Presentation Specification Data 
Manager as the third step. If SAM's default setting is chosen, then SAM assumes, for 
example, that slides are displayed sequentially, each slide has the same duration and the 
first slide starts displaying at the very beginning of the presentation. Therefore, SAM can 

25 calculate the presentation specification Presentation Specification Data automatically and 
does not need to activate the Presentation Specification Data Manager. 

Step 3 - Creating Presentation Specification Data (Presentation Specification 
Data Manager) 

In this step, the Presentation Specification Data Manger is activated to create the 
30 presentation specification Presentation Specification Data. Presentation specification 
Presentation Specification Data is used to synchronize different media within a 
multimedia presentation. It is as simple as timing information when to change to the next 
slide (image, text, or html file) or when to open a window for live media stream, possibly 
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synchronized with the content of time-intrinsic media (such as video or audio clips). With 
Presentation Specification Data Manager, users can decide at which timing point a slide 
or window starts displaying based on the timeline of another playing audio or video clip. 
An example implementation of a video lecture, in which two slide files (formatted as 
HTML files) and a opening of a live streaming source are synchronized to an explanatory 
video clip based on the Presentation Specification Data file shown below: 
sldOO I. htm> 123 1646556 ~ " 



sld002.htm>5464312313 



livesource.s21>7634537334 



In the Presentation Specification Data file, the string before character •>' is the 
indicator or handle of one slide file or a streaming source, while the string after character 
'>' is the exact media time when the slide or streaming source should start displaying. 
Whenever the current media time exceeds one of the media times in the Presentation 
Specification Data file, the corresponding displayed is activated. Since the media times 
are associated with the timeline of one video clip, then the synchronization between the 
video and the slide files is implemented. 

Step 4 - Generating Media Missile Presentation (Document fipnprafnr) 

Finally, the wizard shows the summary information such as the template name of 
the presentation, file names or URLs of selected media, and the Presentation 
Specification Data for synchronization. After clicking Finish button, the Presentation 
Wizard sends all the information to a Document Generator. The Document Generator 
creates a SMIL presentation specification or MM document based on a selected template 
file. In one embodiment, it reads the template file line by line, parsing it, and collecting 
and then placing the corresponding media data, file names, URLs or presentation data 
into a Media Missile structure. 

SAM Co mponent and Screen Implementation Examples 

FIG. 16 illustrates an example of a Preview Manager GUI according to specific 
embodiments of the present invention. This Preview Manager is a single window, split 
into a media preview area and a control area at the top of the window. A text field control 
contains the current medium file location. Users can choose a medium in any way know 
for selecting files or indicating locations or streaming sources (URLs) such as typing a 
new path inside the text field; or pressing the button labeled ... to select a location. The 
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selected path is then shown in the text field. After selection, users can press Preview 
button to start displaying the medium. 

According to specific embodiments, each Presentation Wizard step can be 
implemented as a dialog box. Each dialog box has three buttons in common. Users can 
5 click Back button to return to the previous wizard step or click Next button to go forward 
to the next wizard step. Users can click Cancel button to close the wizard at any step 
without generating a document. 

FIG. 17A-C illustrate examples of choosing various presentation templates 
according to specific embodiments of the present invention. 
10 The Adding and Previewing Media functionality can change based on the 

template selected. FIG. 18A shows the Video Lecture template Add and Preview Media 
screen. Labels such as "Add Video Clip", "Add Slide Files", "Video Duration", and 
"Each Slide Duration" are set based on the media support of this template. Users can 
add, remove, or preview media in this dialog box. Users can click Remove button to 
15 delete selected slide file in the list box. Users can also click Preview button to activate the 
Preview Manager for playing back the selected medium file. Figure 7-19 shows a selected 
HTML slides displayed in the Preview Manager. 

FIG. 19A illustrates the situation when Headline News template is selected in step 
one. Headline News consists of video synchronized with text notations, which come from 
20 direct input instead of text file. Therefore, a dialog box FIG. 19B is provided for direct 
text input and is activated when Add Text Notations button is clicked. 

Based on the template selection, GUIs can also present different Presentation 
Specification Data creation methods. FIG. 19A, for example, shows "Using Presentation 
Specification Data Manager" radio button, which means the wizard will activate the 
25 Presentation Specification Data Manager dialog box as the third step. "Using Default 
Setting" radio button indicates that users just need to type in the duration data. The 
wizard assumes that each slide has the same duration and the first slide starts displaying 
at the very beginning of the presentation. Therefore, the wizard does not need to activate 
the Presentation Specification Data Manager and goes directly to the last step - step 4. 
30 Similarly to Step 2, the functionality of the Presentation Specification Data 

Manager changes based on the template selection in step one. FIG. 20A shows an 
example when users select the Video Lecture template in step one and choose some 
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media files in step two. The "Video Lecture" label is shown in the Presentation 
Specification Data Manager. Labels such as "Video File", "HTML Slide List", "HTML 
Slide Preview", and "HTML Slide File Name" are set based on the media support of 
Video Lecture template. The left part of the Presentation Specification Data Manager is 
the video monitor area. The file path is shown in the Video File text field, and the video is 
rendered and played by a Preview module. When the video starts playing, the 
MediaProgressMonitor activated in step two updates the video Current Time every 50 
milliseconds. 

The right part of the dialog box is the slide timeline setting area. Each slide is not 
only included in the HTML Slide List box, but also shown in the Slide Timeline 
Recording panel in order. Users can click Reset button to reset each TimeLine text field 
in the panel to 0.0 second and click Sort TimeLine button to sort the slides in timeline 
order. FIG. 20B shows the situation when users select the Headline News template in step 



one. 



As shown in Figure FIG. 20A, Slide Timeline Recording panel in Presentation 
Specification Data Manager is the GUI part dynamically generated according to the 
users' selection in step one and two. Each slide file corresponds to one small child panel 
that consists of a File Name text field, a Record button, and a TimeLine text field. Since 
the video timeline is the entire video lecture presentation timeline, users can click Record 
button to set the starting time of the corresponding slide based on the video content. 
When one Record button is clicked, the exact time is shown in the corresponding 
TimeLine text field and the following slide is shown in the HTML Slide Preview area. 

As a farther option, according to specific embodiments, when Users click the Sort 
button, the slides in the Slide Timeline Recording Panel are sorted in timeline order, 
which means the slide with the earliest starting time shows in the first line of the panel 
and so on. Bubble sort is simple and effective; therefore, it can be used as a sort algorithm 
for the timeline. 

Because generating presentation is the last step in the wizard guide, presentation 
information such as template name, selected media files, automatically generated or 
manually generated (by Presentation Specification Data Manager) Presentation 
Specification Data is summarized in the Presentation Information text area. Meanwhile 
users can define the presentation title, author name, and copyright. When Finish button is 
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clicked, the Document Generator will get all the information and generate a presentation 
independently. FIG. 21 illustrates an example last step of a presentation generation. 

SAM can utilize live multimedia streams by using Real-time Transport Protocol 
(RTP) for streaming media objects, and Real-time Transport Control Protocol (RTCP) for 
5 controlling and managing remote stream objects like a traditional VCR does (play, stop, 
and pause). RTP is the Internet standard for the transport of real-time data including 
audio and video stream. RTP can be used for media-on-demand application as well as 
interactive services such as Internet telephony, RTCP is the control protocol designed to 
work in conjunction with RTP. 

10 Further Example Source File 

Below is one example of a SMIL-type presentation specification file that could be 

part of a media missile according to specific embodiments of the present invention and 
could be accessed through a synchronized media player according to specific 
embodiments of the present invention. To display this file on a target machine, for 
15 example, would require players such as video, audio, HTML text, and image players, 

each of which could be resident at a target machine or packaged in a media missile. 

<smil> 
<head> 

<meta name =" author" contents »s. Shira" /> 
20 <meta name= " copyright " contents "SJSU" /> 

<layout> 

<root- layout background- colors "192 , 192 , 192" 

heights" 62 0"/> 
<region id= "videoregion" left="40" top="10" 
25 ~ height = n 100" ~ z-index="0 M 

colors "192, 192, 192 "/> 
<region id="lectureregion" left="10" top="220" 
height="200" z-index="0" 
colors "192, 192, 192"/ > 
30 <region id= "slideregion" left= M 240" top="10 M 

height= "2 00" z-index="0" 
Color="192,192,192"/> 
</layout> 
</head> 
35 <body> 
<par> 

<video src="demo\gm.mpg" regions "videoregion" /> 
<seq> 

<html src="demo\carl .gif " begins "5s" dur="3s" 

40 region=" slideregion" /> 

<html src="demo\car2 .gif " dur="5s 

<html src="demo\car3 .gif " dur= n 5s 

<html src="demo\car6 .gif " dur="5s 

<html src="demo\car5 .gif " dur="5s 
45 <html src= "demo\car4 . gif " dur="5s 

<html src="demo\car5 .gif " dur=" 5s 
</seq> 



width= "1200" 

widths "150" 
background- 

width="450" 
background- 
widths "250" 
background- 



regions "slideregion " / > 
regions M si ideregion " / > 
regions "slideregion " / > 
regions "si ideregion "/> 
regions "slideregion »/> 
regions "si ideregion "/> 
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<seq> 

<html src="demo\gml .html" begin="6s" dur="6s" 

region="lectureregion"/> 
<html src="demo\gm2 .html" dur="6s" region="lectureregion"/> 
<html src="demo\gm3 .html" dur="6s" region="lectureregion"/> 
<html src="demo\gm4 .html" dur="6s" region="lectureregion"/> 
<html src= "demo\gm5.html" dur="6s" region- "lecturereqion" /> 
</seq> 
</par> 
</body> 
</smil> 

Embodiment in a Programmed Information Appliance 

FIG. 22 is a block diagram showing a representative example logic device in 
which aspects of the present invention may be embodied. The invention can be 
implemented in hardware and/or software. In some embodiments of the invention, 
different aspects of the invention can be implemented in either client-side logic or a 
server-side logic. As will be understood in the art, the invention or components thereof 
may be embodied in a fixed media program component containing logic instructions 
and/or data that when loaded into an appropriately configured computing device cause 
that device to perform according to the invention. As will be understood in the art, a 
fixed media program may be delivered to a user on a fixed media for loading in a users 
computer or a fixed media program can reside on a remote server that a user accesses 
through a communication medium in order to download a program component. 

FIG. 22 shows an information appliance (or digital device) 700 that may be 
understood as a logical apparatus that can read instructions from media 717 and/or 
network port 71-9. Apparatus 700 can thereafter use those instructions to direct server or 
client logic, as understood in the art, to embody aspects of the invention. One type of 
logical apparatus that may embody the invention is a computer system as illustrated in 
700, containing CPU 707, optional input devices 709 and 711, disk drives 715 and 
optional monitor 705. Fixed media 717 may be used to program such a system and may 
represent a disk-type optical or magnetic media, magnetic tape, solid state memory, etc.. 
The invention may be embodied in whole or in part as software recorded on this fixed 
media. Communication port 719 may also be used to initially receive instructions that are 
used to program such a system and may represent any type of communication connection. 

The invention also may be embodied in whole or in part within the circuitry of an 
application specific integrated circuit (ASIC) or a programmable logic device (PLD). In 
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such a case, the invention may be embodied in a computer understandable descriptor 
language which may be used to create an ASIC or PLD that operates as herein described. 

Other Embodiments 

The invention has now been described with reference to specific embodiments. 

5 Other embodiments will be apparent to those of skill in the art. In particular, a user digital 
information appliance has generally been illustrated as a personal computer. However, the 
digital computing device is meant to be any device for interacting with a remote data 
application, and could include such devices as a digitally enabled television, cell phone, 
personal digital assistant, etc. 

10 While the forgoing and attached are illustrative of various aspects/embodiments of 

the present invention, the disclosure of specific sequence/steps and the inclusion of 
specifics with regard to broader methods and systems are not intended to limit the scope 
of the invention which finds itself in the various permutations of the features disclosed 
and described herein as conveyed to one of skill in the art. 

15 Furthermore, while the invention has in some instances been described in terms of 

client/server application environments, this is not intended to limit the invention to only 
those logic environments described as client/server. As used herein, "client" is intended 
to be understood broadly to comprise any logic used to access data from a remote system 
and "server" is intended to be understood broadly to comprise any logic used to provide 

20 data to a remote system. 

It is understood that the examples and embodiments described herein are for 
illustrative purposes only and that various modifications or changes in light thereof will 
be suggested by the teachings herein to persons skilled in the art and are to be included 
within the spirit and purview of this application and scope of the claims. 
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WHAT IS CLAIMED: 

1 . A method of accessing streaming media content compri sing: 
initiating a media missile; 

executing a scheduler, said scheduler loading a presentation specification; 
under control of said scheduler, initiating one or more players; 
passing indications of media objects in said media missile to one or more players; and 
at said one or more players, under control of said scheduler, presenting said media 
objects from said media missile. 

2. The method of claim 1 further comprising: 

at said one or more players, retrieving one or more headers from said media missile 

corresponding to a media object indicated for said one or more players; 
at said one or more players, inspecting said one or more headers to determine the 

necessity of downloading lightweight software components from said media missile 

to render the media data; and 
at said one or more players, loading media-specific components at runtime if needed to 

access said media data. 

3. The method of claim 1 wherein said accessing is performed on a mobile platform 
with limited pre-installed software capability, 

4. The method of claim 1 wherein said accessing is performed on a mobile platform 
with limited memory footprint. 

5. The method of claim 1 wherein said accessing enables synchronized presentation 
on a mobile platform. 

6. The method of claim 1 further comprising: 

at said one or more players, setting one or more barrier synchronization primitives. 

7. The method of claim 1 wherein said one or more players plays media data 
according to an enable signal from said scheduler. 
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8. The method of claim 1 wherein said initiating comprises launching a media 
missile from local storage, said media missile including a initial player scheduler able to 
run on a local device's runtime platform. 

9. The method of claim 1 wherein said initiating comprises launching a media 
5 missile from remote storage, said media missile including a initial player able to run on a 

local device's runtime platform. 

10. The method of claim 1 wherein said initiating comprises accessing a presentation 
formatted in a web page. 

11. The method of claim 1 wherein said scheduler is taken from said media missile. 
10 12. The method of claim 1 wherein said scheduler is preinstalled on an access device. 

13. The method of claim 1 wherein said presentation specification is loaded from said 
media missile. 

14. The method of claim 1 wherein one or more of said indications of media objects 
includes a signature. 

15 15. The method of claim 1 wherein said media missile is a document container 
including active content and passive content, said active content comprising machine- 
aware executable components and said passive content comprising presentation 
specification, metadata information, and media data. 

16. . The method of claim 1 wherein said media data can include another media 
20 missile. 

17. The method of claim 15 wherein said active and passive content of said media 
missile can be accessed from a single streaming file location. 

1 8. A method of accessing streaming media content comprising: 
initiating a media missile; 

25 wherein said media missile is a document container capable of holding machine-aware 

active content and passive content in a variety of formats; and 
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passing indications of media objects in said media missile to one or more players. 

19. The method of claim 18 wherein said initiating comprises: 
using a browser to point-and-shoot said media missile. 

20. The method of claim 1 8 wherein said initiating comprises: 

using a command line interface to launch a local self-contained media missile. 

21. The method of claim 1 8 wherein said initiating comprises: 
initiating a player with media missile name as input. 

22. The method of claim 1 8 wherein said initiating comprises: 

wherein said media missile contains a complete executable component and is able to 
load itself and run in accordance with runtime loader characteristics available on a 
particular operating system platform. 

23. The method of claim 18 wherein at least one of said one or more players is 
included in said media missile. 

24. The method of claim 18 wherein at least one of said one or more players is an 
external player. 

25. The method of claim 1 8 wherein said external player is an MPEG player. 



in a 



26. The method of claim 1 8 wherein a hyperlink retrieval of said media missile 
web-enabled page implicitly streams only an initial active machine-aware portion of said 
media missile with the remainder of said media missile streamed under control of said 
initial active machine-aware portion. 

27. The method of claim 1 8 wherein said accessing may be performed on a platform 
with limited pre-installed software capability. 

28. The method of claim 18 wherein said accessing is performed on a platform with 
limited memory footprint. 
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29. The method of claim 1 8 wherein playback of a Media Missile further comprises 
potentially overlapping processes of delivering, assembling, and rendering media missiles 
and their contents. 

30. A method of playing a synchronized multimedia presentation comprising: 
5 initiating a media missile; 

wherein said media missile is a document container capable of holding machine-aware 

active content and passive content in a variety of formats; 
initializing a synchronized multimedia player with a timeline-based scheduler to 

schedule playback according to a presentation specification; and 
10 passing indications of media objects in said media missile to one or more players. 

31. The method of claim 30 wherein said presentation specification is available 
through a web page. 

32. The method of claim 30 wherein said presentation specification is available from 
said media missile. 

15 33. The method of claim 30 wherein said synchronized multimedia player is resident 
on a platform prior to said initiating. 

34. The method of claim 30 wherein said synchronized multimedia player is in said 
active content of said media missile. 

35. The method of claim 30 further comprising: 

20 parsing said presentation specification to build a presentation tree; and 

firing said playing from the top of said tree based on the synchronization 
characteristics of intermediate tree nodes. 

36. The method of claim 35 further comprising: 

for one or more parallel nodes, creating independent execution threads to handle each 
25 parallel task, each independent thread referring to a global time base. 

37. The method of claim 35 further comprising: 
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for one or more sequential nodes, using a single execution thread to step through 
sequential tasks. 

38. The method of claim 35 further comprising: 

upon firing a leaf node, locating a correct media player and media data. 

39. The method of claim 30 fiirther comprising: 

if a link to media data in said presentation specification is a relative path, searching 

said media missile for said media data; and 
if a link to media data in the specification is not a relative path, following an absolute 

path to locate the media data. 

40. The method of claim 39 wherein said absolute path is either specified entirely in 
said link or is initiated from a root of a server from where said media missile was 
streamed. 

41. A method of accessing a multimedia presentation comprising: 

checking a media missile to determine if said media missile contains an executable 

portion for a current platform; 
if said checking is positive, launching machine-aware components in said media 

missile on the fly or as an enhancement of local software to render media data 

within said media missile; and 
if said checking is negative, launching a local media player with default components to 

read media data from said media missile; wherein said media missile is a document 

container capable of holding machine-aware active content and passive content in a 

variety of formats. 

42. The method of claim 4 1 further comprising: 

initializing a synchronized multimedia player with a timeline-based scheduler to 

schedule playback according to a presentation specification; and 
passing indications of media objects in said media missile to one or more players. 

43. The method of claim 42 wherein said player handles both barrier and event 
synchronization with elastic timers. 
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44. The method of claim 42 wherein said player internally handles synchronization 
events according to a presentation specification. 

45. The method of claim 44 wherein a presentation specification can define precise 
timing sequence or a relaxed schedule. 

5 46. The method of claim 44 wherein a playback can be either sequential or concurrent 
(parallel). 

47. A media missile streaming document container comprising: 
a first location for active media content; and 

a second location for passive media content; 
10 wherein said active media content comprises machine-aware executable code and said 

passive media content can comprise presentation specification, metadata 
information, and media data. 

48. A streamable media missile file comprising: 
a first portion for holding executable content; 

15 a second portion for holding a presentation specification; 

a third portion for holding metadata; and 
a fourth portion for holding one or more media data objects. 

49. The device according to claim 47 wherein one or more of said media data objects 
can include another media missile, providing a nested format. 

20 50. The device according to claim 47 wherein said media missile file can be managed 
at a server and distributed from a server as a single file. 

51. The device according to claim 47 wherein said media missile file is generated on 
the fly at a server though is delivered to a user as a single streaming file. 

52. The device according to claim 47 further comprising: 

25 a first length field indicating if machine-executable code is present and the length of 

said code if present; and 
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a first continuation field indicating presence of other portions in said media missile 
file. 

53. The device according to claim 52 wherein: 

if said first continuation field is logical true and said first length field indicates a zero 
length, said media missile begins with said presentation specification; 

if said first continuation field is logical false and said first length field indicates a zero 
length, said media missile begins with said metadata; and 

if said first continuation field is logical false and said first length field indicates a 
negative value, said media missile comprises pure media data. 

54. The device according to claim 47 further comprising: 

a second length field indicating if a presentation specification is present and the length 

of said presentation specification if present; and 
a second continuation field indicating presence of other portions in said media missile 

file. 

55. The device according to claim 54 wherein: 

if said second continuation bit field is logical true, then metadata information follows 

said presentation specification; and 
if said second continuation bit field is logical false, then media data follows said 

presentation specification. 

56. The device according to claim 54 further comprising: 

a third length field indicating if metadata is present and the length of said metadata if 
present; and 

a third continuation field indicating whether data after said metadata is another media 
missile or media data. 

57. The device according to claim 56 wherein: 

said metadata comprises one or more media object length fields indicating length of 
one or more media data objects in said media missile. 

58. The device according to claim 57 wherein: 
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wherein said one or more media objects are stored in sequence according to a sequence 
of said media data length field(s). 

59. The device according to claim 56 wherein one or more media data length fields 
includes a field indicating whether a corresponding object is a resource locator or is a 

5 payload. 

60. A method of assisting a user in creating a synchronized multimedia presentation 
comprising: 

prompting said user to select a template for said presentation; 
prompting said user to select one or more media data objects; 
10 prompting said user to select an option for generating presentation specification data; 

and 

preparing final output, including a presentation specification. 

61. The method according to claim 60 wherein said final output comprises a 
multimedia missile that can include media objects, a presentation specification, metadata, 

15 and active components in a container deliverable as a stream from a single streaming 
source location. 

62. The method according to claim 60 wherein said final output comprises a 
presentation specification in a standard descriptive tagged language. 

63. The method according to claim 60 wherein said method enforces generation of a 
20 well-formed synchronized multimedia presentation. 

64. The method according to claim 60 wherein final output can include both precise 
and elastic synchronization primitives. 

65. The method according to claim 60 further comprising':' 

prompting said user to modify duration settings for media data objects; and 
25 prompting said user to use a presentation metadata manager to customize a 

presentation timeline. 

66. The method according to claim 65 further comprising: 
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displaying the video and con-esponding slides at a particular point to an user to assist 
creating a presentation. 

67. The method according to claim 61 further comprising: 

prompting said user to select whether to keep the final output locally or upload it to a 
specific hosting server. 

68. The method according to claim 60 further comprising: 

accepting presentation specification and media data as input to said authoring 
method. 

69. The method according to claim 60 further comprising: 

creating a top-down tree representation of synchronization primitives wherein each 
intermediate node is either a parallel or sequential synchronization primitive; and 

determining required fields according to a media missile specification for terminal leaf 
nodes in said tree. 

70. The method according to claim 69 further comprising: 

attaching fields to individual media data specified in a presentation so that said 
individual media have the format of a media missile with media data only. 

71. The method according to claim 70 further comprising: 

using a depth-first tree traversal to calculate and create optional fields based on user 
input; and 

using a breadth-first tree traversal to traverse the entire tree and create the final media 
missile. 

72. A system for streaming synchronized multimedia presentations comprising: 
a real-time data source providing a plurality of streaming media objects; 

a data store for holding and retrieving presentation specifications or media missiles; 
and 

a repository manager that transmit a streaming media missile to a target device, where 
the media missile can include packaged indications from said real-time data source, 
can include static media objects, and can include presentation specification, 
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73. The system of claim 72 further comprising: 
an authoring manager for receiving authoring specifications for a streaming 

presentation. 

74. The system of claim 72 wherein said authoring manager accepts indications 
provided by an author and using said indications prepares a well-formed streaming 
presentation in accordance with a predefined descriptive tagged language. 

75. The system of claim 72 wherein said repository manager handles transmission of 
executable components as said components are needed and are requested by a 
synchronized multimedia player said for accessing a portion of said streaming 
presentation. 

76. The system of claim 75 wherein said repository manager and said player are able 
to begin pre-fetching delay-sensitive media data. 

77. The system of claim 72 wherein data is delivered from said repository manager in 
a container that provides indications for machine-aware executable code, media data, and 

15 associated information. 

78. The system of claim 77 wherein said container can include one or more additional 
containers in a nested structure. 

79. The system of claim 77 wherein said container holds all active and passive content 
necessary to play a synchronized multimedia presentation. 

20 80. The system of claim 72 wherein said authoring manager understands various 
language specifications, formats, and templates in synchronized multimedia. 

81. The system of claim 72 wherein said authoring manager includes a wizard that 
guides a user to compose a synchronized presentation and retrieve media data from a 
media repository manager to produce a media missile. 

25 - 82. The system of claim 72 further comprising a synchronized multimedia player 
including an internal scheduler to process delay-sensitive data. 
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83. The system of claim 82 wherein said synchronized multimedia player can present 
presentations in accordance with SMIL presentation data. 
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Example METADATA block FORMAT 



{ // metaDataBlock 
{ bitl data_or_missile 
block is data 

int63 metaDataBlkLen; // si zeof (metaDataBlock) ; 

intl28 signature; // 0x03010101 => player version 3.01, 

// f ileFormatld 1, version 1 
bitl payload_or_url ; 
int63 def aultUrlLen; 
{ bitl payload_or_url; 
int63 f ilenameLen; 

// always 0; => payload 



// in this example always 0; 



// always 1; => URL 

// always 0; => payload 



next 



bitl payload_or_url ; 
int63 f ileDataLen; 
} f ileNameDataLenStruct [numberOf Files] ; 
// mediaDataBlock 
{ char defaultUrl [defaultUrlLen] ; 



} 



{ 



} 



char filename [f ilenameLen] ; 
char fileContent [f ileDataLen] ; 
} f ileNameDataStruct [numberOf File] ; 
mediaDataBlock; 
// ending 
bitl p_blk_exist; 
int63 eLength; } 
//be aware the byte order difference in file vs in memory 
// e.g. int value = READ ( ) ; File Hex Data "12 34 56 78"; 
// value in Hex is 0x78563412; 
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EXAMPLE METADATA BLOCK CONTENTS 
Assumed values 

Default URL is "http : //www. abcCo . com" strlen 20 

c:\webAds\s21WebAds.htm nameLen 13, size 2562 bytes // the 

main page 

c:\webAds\otherLinks.htm nameLen 14, size 1548 bytes 

c:\webAds\logo.gif nameLen 8, size 5874 bytes 

c:\webAds\sponsor\abcCo.jpg nameLen 17, size 18745 bytes 

Metadata Media Missile values 



bitl 


da t a_or_mi s s i 1 e 


II 


0 


int63 


metaDataBlkLen; 


II 


(64/8) * (4+4*2) = 96 bytes 


int!28 signature; 


II 


0x03010101 


DILI 


pay lUau \Ji. ux i , 


II 


1 




dpf aultUrlLen • 


II 


20 


bitl 

U J- L. X 


navload or url • 


II 


0 


•5 nt- fil 

lllL 0 J 


L ixciiaincuciii , 


II 


13 


DILI 


pay ludu ui ui j. , 


II 


0 


int 63 


fileDataLenl ; 


II 


2562 


bitl 


payload or url ; 


II 


0 


1I1LO J 


f i 1 pnampl,pn9 • 
i. x x ciiaiucucii^ , 


II 


. 14 


bitl 


payload or url; 


II 


0 


int 63 


f i 1 eDataLen2 : 


II 


1548 


bitl 


navload or url : 


II 


0 


int63 


f ilenameLen3 ; 


II 


8 


bitl 


payload_or_url ; 


If 


0 


int63 


f ileDataLen3 ; 


II 


5874 


bitl 


payload_or_url ; 


II 


0 


int63 


f ilenameLen4 ; 


II 


17 


bitl 


payload_or_url ; 


II 


0 


int63 


f ileDataLen4 ; 


II 


18745 


// At 


96 bytes; this is the 


media data 


char 


defaultUrl [20] 


II 


" ht tp : / /www . abcCo . com" 


char 


f ilenamel [13] 


II 


" s 2 1 WebAds . htm 11 ( s imp 1 e 




encryption) 






char 


fileDatal [2562] 


II 


s 2 1 WebAds . htm 1 s data 


char 


f ilename2 [14] 


II 


" otherLinks . htm" ( s imple 




encryption) 




otherLinks.htm' s data 


char 


fileData2 [2562] 


II 


char 


f ilename3 [8] 


II 


"logo.gif" 


char 


fileData3 [2562] 


II 


logo.gif's data (simple 




encryption) 




11 sponsor \abcCo. jpg " 


char 


f ilename4 [17] 


II 


char 


f ileData4 [2562] 


II 


sponsor\abcCo . j pg ' s data 




encryption) 






bitl 


contination_bit 


II 


0 


int63 


eLength // 0 







FIG. 10 



BNSDOCID: <WO 0126378A1_L> 



WO 01/26378 



8/17 



PCT/US00/27634 



char 
bitl 
int63 
char 
bitl 
int€3 



MEDIA MISSILE BLOCK CONTENTS EXAMPLE 

executablecode [nnnnnn] ; // Executable code 

// 1 

// PPPP 

II Presentation Specification 



(64/8) * (4+4*2) 
// 0x03010101 
1 
20 
0 
13 
0 

2562 
0 
14 
0 

1548 
0 
8 
0 

5874 
0 
17 
0 



96 bytes 



P_spec_contination_bit ; 
pLength ; 

presentation_spec Ipppp] 

data_or_missile // 

metaDataBlkLen; // 
intl28 signature; 

bitl payload_or_url; // 

int63 defaultUrlLen; // 

bitl payload_or_url; // 

int63 filenameLenl; // 

bitl payload_or_url; // 

int63 fileDataLenl; // 

bitl payload_or_url; // 

int63 f ilenameLen2; // 

bitl payload_or_url; // 

int63 f ileDataLen2; // 

bitl payload_or_url; // 

int63 f ilenameLen3 ; // 

bitl payload_or_url; // 

int63 f ileDataLen3; // 

bitl payload_or_url; // 

int63 f ilenameLen4 ; // 

bitl payload_or_url; // 

int63 f ileDataLen4 ; // 18745 

// 

// At eLength + pLength plus 104 
that 

//if the data_or_missile bit = 1 # this portion could begin another 

// media missile. In that case, the words after the signature block 

above will 
// have the form: 

// bitl payload_or_url; // o 

// int63 MediaMissileLenl; // mmmm 

II and the media data will include the object 

// 
// 

char 
char 

char 
char 

char 
char 
char 

char 
char 

bitl 



bytes; this is the media data; or note 



char mediamissile [mm/nm] 



def aultUrl [20] 
f ilenamel [13] 

encryption) 
fileDatal [2562] 
f ilename2 [14] 

encryption) 
fileData2 [1548] 
f ilename3 [8] 
f ileData3 [5874] 

encryption) 
filename4 [17] 
fileData4 [18745] 

encryption) 
continuation bit 



int63 eLength // nnnnnn 



II " h t tp : / /www . abc Co . com » 
// "s21WebAds.htm" (simple 

// s21WebAds.htm' s data 
// "otherLinks.htm 1 ' (simple 

// otherLinks.htm' s data 
// "logo.gif" 

// logo.gif 's data (simple 

// " sponsor\abcCo . j pg" 
// sponsor \abcCo. jpg 1 s 

// 0 

FIG. 11 
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;S21PackData.dat] 



000000 


po 


00 


00 


00 


00 


00 


00 


00 


01 


01 


01 


03 


00 


00 


00 


00 


AAA Art f\ 

000010 


00 


00 


00 


00 


00 


00 


00 


00 


00 


00 


00 


00 


00 


00 


00 


80 


000020 


OF 


00 


00 


00 


00 


00 


00 


60 


OF 


00 


00 


00 


00 


00 


00 


00 


000030 


00 


03 


00 


00 


00 


00 


00 


00 


OC 


00 


00 


00 


00 


00 


00 


00 


000040 


03 


49 


00 


00 


00 


00 


00 


00 


08 


00 


00 


00 


00 


00 


00 


00 


000050 


31 


13 


00 


00 


00 


00 


00 


00 


BF B8 


BE A9 


AD 


Al 


A5 


A2 


000060 


AB 


FE 


FD 


E2 


A4 


B8 


Al 


BF 


B8 


BE 


A9 


AD 


Al 


A5 


A2 


AB 


000070 


FE 


FD 


E2 


A4 


B8 


Al 


FO 


A4 


B8 


Al 


AO 


F2 


CI 


C6 


CI 


CS 


000080 


F0 


A4 


A9 


AD 


A8 


F2 


CI 


C6 


FO 


B8 


A5 


B8 


AO 


A9 


F2 


9F 


000090 


B8 


BE 


A9 


AD 


Al 


A5 


A2 


AB 


EC 


9C 


AO 


AD B5 


A9 


BE 


FO 


OOOOaQ 


E3 


B8 


A5 


B8 


AO 


A9 


F2 


CI 


C6 


FO 


80 


85 


82 


87 


EC 


9E 


OOOObO 


89 


80 


Fl 


9F 


B8 


B5 


AO 


A9 


9F A4 


A9 


A9 


B8 


EC 


84 


9E 


OOOOcO 


89 


8A 


Fl 


EE 


BC 


AD 


AB 


A9 


E2 


AF 


BF 


BF 


EE 


EC 


98 


95 


OOOOdO 


9C 


89 


Fl 


EE 


B8 


A9 


B4 


B8 


E3 


AF 


BF 


BF 


EE 


F2 


CI 


C6 


OOOOeO 


CI 


C6 


F0 


E3 


A4 


A9 


AD 


A8 


F2 


CI 


C6 


CI 


C6 


FO 


AE 


A3 


OOOOf 0 


A8 


B5 


EC 


AE 


AB 


AF 


A3 


AO 


A3 


BE 


Fl 


EE EF FC 


FC 


FC 


000100 


FC 


FC 


FC 


EE 


EC 


B8 


A9 


B4 


B6 


Fl 


EE 


EF 


8A 


8A 


8A 


8A 


000110 


8A 


8A 


EE 


EC 


AO 


A5 


A2 


A7 


Fl 


EE EF 


8A 


8A 


8A 


8A 


8A 


000120 


6A 


EE 


EC 


BA 


AO 


AS 


A2 


A7 


Fl 


EE EF 


8A 


8A 


8A 


8A 


8A 


000130 


8A 


EE 


EC 


AD 


AO 


A5 


A2 


A7 


Fl 


EE 


EF 


8A 


8A 


8A 


8A 


8A 


000140 


8A 


EE 


EC 


81 


8D 


9E 


8B 


85 


82 


9B 


85 


88 


96 


84 


Fl 


EE 


000150 


FC 


EE 


EC 


80 


89 


8A 


98 


81 


8D 


9E 


8B 


85 


82 


Fl 


EE 


FC 


000160 


EE 


EC 


81 


8D 


9E 


8B 


85 


82 


84 


89 


85 


8B 


84 


98 


Fl 


EE 


000170 


FC 


EE 


EC 


98 


83 


9C 


81 


8D 


9E 


8B 


85 


82 


Fl 


EE 


FC 


EE 


oooieo 


F2 


CI 


C6 


CI 


C6 


FO 


A8 


A5 


BA 


EC 


AD AO 


AS 


AB 


A2 


Fl 


nnni on 
00U19U 


L£ 


At 


X o 


A2 


bo 


X o 
A9 


DC* 


Eh 


F2 


CI 


C6 


EC 


EC 


FO 


x n* 
AF 
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BE 


F2 


CI 


C6 


EC 


EC 


FO 


B8 
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AO 
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EC 
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B1 



prompt a user to select a presentation 
template 



4 



B2 



prompt a user to select one or more media 
. data objects 



B3 



prompt a user to select an option for 
generating presentation specification data 



B4 



preparing final output data including a 
presentation specification 



B5 



if indicated by user, packaged final output 
data, including media objects and indications 
of media objects, presentation specification, 

metadata, and active components into a 
media missile container streamable from a 
single location 
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For Help, press F1 



Q Form Designer - PreviewManager 




FIG. 16 Preview Manager GUI 
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Eg SAT Presentation Wizard - Choose Templa 



This Presentation Wizard guide you through the process of creating a . multimedia presentation 
Layout used in template Template: 



Video Lecture 



These templates inlcude predefined track types 
and layouts setting. 



Play a video with slides. 

Suggested application: 
Video Presentation. 



Nsxt> I I Cancel 



FIG. 17A Choosing a Video Lecture Template 



Eg SAT Presentation Wizard - C hoose Template 



This Presentation Wizard guide you through the process of creating a . multimedia presentation 



SMIL Layout used in template 



Template: 



Headline News 



These templates inlcude predefined track types 
and layouts selling. 



Play a video with text streams. 

Suggested application: 
News Broadcasting. 



Next> 



FIG. 17B Choosing a Headline News Template 
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3 SAT Presentation Wizard — Choose Tempi ate 



This Presentation Wizard (juiiie you through the process of c» eating a murtimedia presentation 



Layout used in template 



Template: 
| Slide Show 



3 




These templates inlcude predefined track types 
and layouts setting. 



Play an image with a soundtrack. 

Suggested application: 
Family vacation slideshow, 
product demo, 

corporate training presentation. 



Cancel 



FIG. 17C Choosing a Slide Show Template 



fegSAT Presentation Wizard — Add Media 



Video Lecture 
Add Video Clip: 



|F^master\dance.avi 



Preview 



Add Slide Files: 



F:\master\docidance\sld001 .htm 
F:\ma ster\doc\dance\sid002.htm 



I I Add- 



i Remove 



Preview 



Set Time for Media: 
•» Using Melad< 

O Using Default Setting 

Video Duration: 
Each Slide Duration: 



seconds 
seconds 



{ <Back | Next> 



Cancel [ 



FIG. 18A Video Lecture Add And Preview Media 
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3 SAT Preview M anager 



F:\mastertdoc\dance\sld001 .htm | j Preview | 



Dance - Slide One 
•romantic 
•breath-taking 




Close 



FIG. 18B Video Lecture Template Preview Selected Slides 



Eg SAT Presentation Wizard - Add Media 



Headline News 
Add Video Clip: 

| F:\ma stertyideo.avi 



| Preview"] 



Add Text Notations: 



First JumplDBeautiful Turn'DWoo! 
Into the water..OCool...DBravolil 



Remove [ 



Preview 



Set Time tor Media: 

<§> Using Metadata Manager 

O Using Default Setting 
Video Duration: 
Each Text Duration: 



seconds 
seconds 



<Back 



NexO 



Cancel | 



FIG. 19A Headline News Template - Add And Preview Media 
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i 



Wi Add Text Notation for Headline News 



Add Text Notation for Headline News: 



First Jump! 
Beautiful Turn! 
Woo!| 



| OK 



FIG. 19B Add Text Notation For Headline News 



EgSAT Presentation Wizard — Generate Metadata 



Video Lecture 

Video File !F:\mastertdance.avl 




«C J 



Current Time/ Total Time: 11 6.0/1 8.0 



HTML Slide List: 



HTML Slide Preview: 



1 F:\masteittocttanceU. *\ Slide TWO^ 

] , aiding 



Sort Time Una 



HTML Slide File Name 


TimeLrne 


:Fr^aster\doc\danceiisTd001 .h 


Record !0.0 






.F:\master\doc\dancetsld002.h 


Record | :15.8 





<8ack 



Next> 



J I Cancel | 



FIG. 20A Video Lecture Template - Create Presentation Specification 

Data 
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gjjSAT Pres entation Wizard - Generate Metadata 



Headline News 

Video File: iFr\mastertvideo.avi 





u 



Current Time! Total Time: [1 4.6tf 0.0 



Text Notation List- 



First Jump! □Beautiful . ▼ I 



Text Notation Preview: 



into the water.. 
Cool... 



Sort Time Line 















r irstJumpl 










3eautlful Tumi 
A/OOl 


Ftecord 


jW ~J 












J 


Into the water! 










■Cool... 
Bravolll 


Record 


i?4.T j 













Next> | | Cancel 



FIG. 20B Headline News Template - Create Presentation Specification 

Data 



\ SAT Presentation Wizard - Finish 



You have completed us inn the Presentation Wizard. Your multimedia presentation will now be created. 
Presentation Information: 



Video Lecture: 

Video File: F:\mastertdance.avi 
Video Duration: 18.0sec 

Metadata: manually generated 

First HTML Slide File Begin at O.Oset 

HTML Slide File and Duration: 
F:\mastertdoc\d ance\sld001.htm 15.8sec 
F:\master\doc\dance\sld002.htm 2.1 999998sec 



Presentation Title: 
[videoLecturel 



Author Name: 
|Ying Wang 



Copyright: 



SJSU 



<Back Finish 



Cancel 



FIG. 21 Video Lecture Template - Generate Document 
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